Presenting a View within a Three Dimensional Scene

ABSTRACT

Presenting a view based on a virtual viewpoint in a three dimensional (3D) scene. The 3D scene may be presented by at least one display, which includes displaying at least one stereoscopic image of the 3D scene by the display(s). The 3D scene may be presented according to a first viewpoint. A virtual viewpoint may be determined within the 3D scene that is different than the first viewpoint. The view of the 3D scene may be presented on the display(s) according to the virtual viewpoint and/or the first view point. The presentation of the view of the 3D scene is performed concurrently with presenting the 3D scene.

PRIORITY INFORMATION

This application is a Continuation of U.S. patent application Ser. No.14/268,613 filed May 2, 2014, titled “Presenting a View within a ThreeDimensional Scene,” whose inventors are Michael A. Vesely and Alan S.Gray, which is a Continuation of U.S. patent application Ser. No.12/797,958, filed on Jun. 10, 2010, issued May 6, 2014 as U.S. Pat. No.8,717,360, titled “Presenting a View within a Three Dimensional Scene,”whose inventors are Michael A. Vesely and Alan S. Gray, which claimsbenefit of priority of U.S. Provisional Application Ser. No. 61/299,538,filed on Jan. 29, 2010, titled “Stereo Multiview Device,” whose inventoris Michael Vesely, all of which are hereby incorporated by reference intheir entirety as though fully and completely set forth herein.

FIELD OF THE INVENTION

The present invention relates to the field of three dimensionalgraphics, and more particularly to a system and method for presenting aview within a three dimensional scene.

DESCRIPTION OF THE RELATED ART

Three dimensional (3D) capable computing devices and real-timecomputer-generated 3D computer graphics have been a popular area ofcomputer science for the past few decades, with innovations in visual,audio and tactile systems. Much of the research in this area hasproduced hardware and software products that are specifically designedto generate greater realism and more natural computer-human interfaces.These innovations have significantly enhanced and simplified the user'scomputing experience.

However, additional tools and improvements to the realm of 3D systemsare desired.

SUMMARY OF THE INVENTION

Various embodiments are presented of a system and method for presentinga view based on a virtual viewpoint within a 3D scene.

A 3D scene may be presented by at least one display. More particularly,one or more stereoscopic images of the 3D scene may be presented by thedisplay(s), e.g., by one or more stereoscopic displays. The 3D scene maybe presented according to a first viewpoint. For example, the firstviewpoint may be based on an eyepoint of a user viewing the 3D scene. Inone embodiment, the method may include determining the first viewpoint,e.g., by determining the eyepoint of the user viewing the 3D scene. Themethod may determine the eyepoint of the user using various techniques,such as a position input device (e.g., glasses which provide eyepointposition information), triangulation, head/eye tracking, etc.Accordingly, the 3D scene may be rendered such that user can view the 3Dscene with minimal distortions (e.g., since it is based on the eyepointof the user).

As indicated above, the 3D scene may be presented by a single display ora plurality of displays. In one embodiment, the 3D scene may bepresented by a vertical display and a horizontal display. For example,the vertical display may present a first stereoscopic image, e.g.,according to a vertical perspective, and the horizontal display maypresent a second stereoscopic image, e.g., according to a horizontalperspective. These two stereoscopic images may form the 3D scene. Infurther embodiments, the two displays may be joined by a curvilinear orblending display, which may also present a stereoscopic image. Thestereoscopic image of the blending display may operate to blend thestereoscopic images of the vertical and horizontal displays. Othernumbers and types of displays are contemplated for presenting the 3Dscene.

At least a portion of the 3D scene may be presented in “open space” infront of or otherwise outside of the at least one display. Thus, asindicated above, at least a portion of the 3D scene may appear as ahologram above the surface of the display. Thus, the user can directlyinteract with objects (displayed virtual objects) in the open spacebecause they co-inhabit physical space proximate to the user. The innervolume is located behind the viewing surface, and portions of the 3Dscene within this inner volume appear “inside” the physical viewingdevice. Thus, objects of the 3D scene presented within the inner volumedo not share the same physical space with the user, and the objectstherefore cannot be directly manipulated by hands or hand-held tools.That is, objects displayed within the inner volume may be manipulatedindirectly, e.g., via a computer mouse or a joystick.

The method may further include determining a virtual viewpoint withinthe 3D scene, where the virtual viewpoint is different than the firstviewpoint. For example, the virtual viewpoint within the 3D scene may beselected or otherwise specified by a user, e.g., the user viewing the 3Dscene. In one embodiment, the user may manipulate a handheld device,such as a stylus, to specify the virtual viewpoint within the 3D scene.For example, the user may use the handheld device to specify a viewpointwithin the 3D scene by positioning the handheld device in the open spaceof the presented 3D scene. Thus, the user input may be “open space userinput” to specify the virtual viewpoint.

The virtual viewpoint may be the viewpoint or perspective of a virtualobject displayed in the 3D scene. For example, a virtual camera may bedisplayed in the 3D scene, and the user may be able to manipulate thevirtual camera in the 3D scene (e.g., using a stylus in open space asindicated above) to specify the virtual viewpoint within the 3D scene.Alternatively, or additionally, the virtual object may be a magnifyingglass or other type of tool or displayed object. In further embodiments,the virtual object may be a character or entity in the scene (e.g., aperson or bird flying through the 3D scene) and the virtual viewpointmay correspond to the eyepoint of the virtual entity within the 3Dscene.

In addition to determining the virtual viewpoint, a field of view of thevirtual viewpoint may be determined. For example, the field of view maybe a wide angle field of view (e.g., corresponding to a wide angle lensof a virtual camera) or a telescopic field of view (e.g., correspondingto a telescopic lens of a virtual camera). Additionally, a magnificationfor the virtual viewpoint may be determined. The field of view and/ormagnification may be determined based solely on the virtual viewpoint,or may be determined based on other factors, e.g., user input. Forexample, the user may specify a telescopic lens or a wide angle lens.The user may also be able to select or specify a specific magnification(e.g., of the telescopic lens). Thus, in some embodiments, the field ofview and/or magnification may be determined based on user input.

In some embodiments, the 3D scene may indicate the virtual viewpoint.For example, the displayed 3D scene may illustrate the frustum of theview of the 3D scene within the 3D scene. As one particular example, thevirtual camera object displayed in the 3D scene may also indicate thefrustum of view of the virtual object within the 3D scene.

A view of the 3D scene may be presented on the at least one displayaccording to the virtual viewpoint. Said another way, the view of the 3Dscene may be rendered from the perspective of the virtual viewpointspecified above. Note that the view of the 3D scene may be presentedconcurrently with the presentation of the 3D scene. Where the field ofview and/or magnification is determined above, the presentation of theview of the 3D scene may be based on the determined (or specified) fieldof view and/or magnification. For example, if a magnification isspecified, the view of the 3D scene may be magnified at thatcorresponding magnification. Thus, the view of the 3D scene may bemagnified compared to a corresponding portion of the 3D scene.

The view of the 3D scene may also be based on the first viewpoint (thatcorresponds to the eyepoint of the user). For example, the view of the3D scene may be presented such that it is easy to see based on the firstviewpoint (e.g., where it is perpendicular to the line of sight of theuser viewing the 3D scene). The view of the 3D scene may be presented inany number of ways. For example, the view of the 3D scene may bemonoscopic or stereoscopic, as desired.

Additionally, the view of the 3D scene may be presented on the samedisplay as the 3D scene or on different displays. For example, the viewof the 3D scene may be presented within the 3D scene. In other words,the view of the 3D scene may be presented on the same display, or on thesame plurality of displays, which are used to present the 3D sceneitself. Alternatively, or additionally, the view of the 3D scene may beprovided on a separate display, e.g., which is dedicated to the view ofthe 3D scene. Thus, in one embodiment, the at least one display (whichis used to present the 3D scene and the view of the 3D scene) mayinclude a first one or more displays and a second one or more displays,and the 3D scene may be provided via the first one or more displays andthe view of the 3D scene may be presented via the second one or moredisplays.

Additionally, the view of the 3D scene may be presented at a positionand orientation which differs from that of the display providing theview of the 3D scene. For example, the display may have a horizontalposition and the view of the 3D scene may be presented on a virtualdisplay which faces (or is perpendicular to) the eyepoint of the userviewing the 3D scene. The view of the 3D scene (or virtual display) maybe positioned in open space of the 3D scene, e.g., in open spacerelative to the display providing the view of the 3D scene.

The method may further include moving the virtual viewpoint, e.g., inresponse to user input. The movement of the virtual viewpoint may beperformed similarly to the specification of the virtual viewpointdescribed above. For example, the user may select the virtual objectspecifying the virtual viewpoint (e.g., the virtual camera object) andmay change the position and/or orientation of the virtual object in the3D scene to move the virtual viewpoint from an original position and/ororientation to a second position and/or orientation.

The movement of the virtual viewpoint may be discrete (e.g., immediatelyfrom the original to the second) or may be a continuous movement. Ineither case, the view of the 3D scene may be updated according to themovement, e.g., discretely or smoothly. In the continuous movement, the3D scene may show the continuous change in viewpoint from the originalviewpoint to the finally specified viewpoint. Thus, the determination ofa virtual viewpoint and the presentation of the view of the 3D scene maybe performed a plurality of times throughout presentation of the 3Dscene, and in particular, throughout movement or change of the virtualviewpoint.

The presentation of the view of the 3D scene may be performed at a framerate or quality which is comparable to the presentation of the 3D scene.Additionally, or alternatively, the view of the 3D scene may be providedat a frame rate or quality which is above a certain threshold. Thisquality or speed may ensure that the view of the 3D scene is provided ina smooth manner, e.g., at a rate where the user does not notice movementartifacts in the view of the 3D scene, such as choppiness or glitching.This quality of presentation may apply when the virtual viewpoint isstatic (e.g., when objects are moving within the view of the 3D scene)or when the virtual viewpoint is being moved (e.g., the movement of thevirtual viewpoint described above).

The 3D scene may be updated based on changes of the first viewpoint(e.g., corresponding to changes of the eyepoint of a user). For example,the user may move his head, thereby changing the location of hiseyepoint. Accordingly, a next viewpoint (which corresponds to the user'snew eyepoint) may be determined after displaying the 3D scene and theview of the 3D scene. Based on this next viewpoint, the 3D scene may beupdated and an updated stereoscopic image of the 3D scene may beprovided by the display(s). Where the view of the 3D scene is also basedon the viewpoint corresponding to the eyepoint of the user, the view ofthe 3D scene may be correspondingly updated as well.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when thefollowing detailed description of the preferred embodiment is consideredin conjunction with the following drawings, in which:

FIGS. 1 and 2 illustrate exemplary systems configured to implementvarious embodiments;

FIGS. 3A and 3B illustrate exemplary horizontal and vertical perspectiveprojections, according to some embodiments;

FIGS. 4A and 4B illustrate an exemplary horizontal display with acorresponding horizontal projection, according to some embodiments;

FIGS. 5A and 5B illustrate exemplary view volumes of a horizontalprojection, according to some embodiments;

FIG. 6 is a flowchart diagram illustrating one embodiment of a methodfor presenting a view of a 3D scene based on a virtual viewpoint;

FIG. 7 illustrates an exemplary horizontal display presenting anexemplary 3D scene and view of the 3D scene, according to oneembodiment; and

FIGS. 8A-16B are exemplary illustrations of the described embodiments.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and are herein described in detail. It should beunderstood, however, that the drawings and detailed description theretoare not intended to limit the invention to the particular formdisclosed, but on the contrary, the intention is to cover allmodifications, equivalents and alternatives falling within the spiritand scope of the present invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE INVENTION Incorporation by Reference

The following references are hereby incorporated by reference in theirentirety as though fully and completely set forth herein: U.S. patentapplication Ser. No. 11/098,681 (U. S. Patent Publication No.2005/0219694), titled “Horizontal Perspective Display”, filed on Apr. 4,2005. U.S. patent application Ser. No. 11/141,649 (U.S. PatentPublication No. 2005/0264858), titled “Multi-plane HorizontalPerspective Display”, filed on May 31, 2005.

Terms

The following is a glossary of terms used in the present application:

Memory Medium—any of various types of memory devices or storage devices.The term “memory medium” is intended to include an installation medium,e.g., a CD-ROM, floppy disks 104, or tape device; a computer systemmemory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM,Rambus RAM, etc.; a non-volatile memory such as a Flash, magnetic media,e.g., a hard drive, or optical storage; registers, or other similartypes of memory elements, etc. The memory medium may comprise othertypes of memory as well or combinations thereof. In addition, the memorymedium may be located in a first computer in which the programs areexecuted, or may be located in a second different computer whichconnects to the first computer over a network, such as the Internet. Inthe latter instance, the second computer may provide programinstructions to the first computer for execution. The term “memorymedium” may include two or more memory mediums which may reside indifferent locations, e.g., in different computers that are connectedover a network.

Carrier Medium—a memory medium as described above, as well as a physicaltransmission medium, such as a bus, network, and/or other physicaltransmission medium that conveys signals such as electrical,electromagnetic, or digital signals.

Computer System—any of various types of computing or processing systems,including a personal computer system (PC), mainframe computer system,workstation, network appliance, Internet appliance, personal digitalassistant (PDA), television system, grid computing system, or otherdevice or combinations of devices. In general, the term “computersystem” can be broadly defined to encompass any device (or combinationof devices) having at least one processor that executes instructionsfrom a memory medium.

Viewpoint—this term has the full extent of its ordinary meaning in thefield of computer graphics/cameras. For example, the term “viewpoint”may refer to a single point of view (e.g., for a single eye) or a pairof points of view (e.g., for a pair of eyes). Thus, viewpoint may referto the view from a single eye, or may refer to the two points of viewfrom a pair of eyes. A “single viewpoint” may specify that the viewpointrefers to only a single point of view and a “paired viewpoint” or“stereoscopic viewpoint” may specify that the viewpoint refers to twopoints of view (and not one). Where the viewpoint is that of a user,this viewpoint may be referred to as an eyepoint (see below). The term“virtual viewpoint” refers to a viewpoint from within a virtualrepresentation or 3D scene.

Eyepoint—the physical position of a single eye or a pair of eyes. Aviewpoint (as defined above) may correspond to the eyepoint of a person.For example, a person's eyepoint has a corresponding viewpoint.

Vertical Perspective—a perspective which is rendered for a viewpointwhich is substantially perpendicular to the display surface.“Substantially perpendicular” may refer to 90 degrees or variationsthereof, such as 89 and 91 degrees, 85-95 degrees, or any variationwhich does not cause noticeable distortion of the rendered scene. Avertical perspective may be a central perspective, e.g., having a single(and central) vanishing point. As used herein, a vertical perspectivemay apply to a single image or a stereoscopic image. When used withrespect to a stereoscopic image (e.g., presenting a stereoscopic imageaccording to a vertical perspective), each image of the stereoscopicimage may be presented according to the vertical perspective, but withdiffering single viewpoints.

Horizontal Perspective—a perspective which is rendered from a viewpointwhich is not perpendicular to the display surface. More particularly,the term “horizontal perspective” refers to a perspective which isrendered using a substantially 45 degree angled render plane inreference to the corresponding viewpoint. The rendering may be intendedfor a display which may be positioned horizontally (e.g., parallel to atable surface or floor) in reference to a standing viewpointperspective. “Substantially 45 degrees” may refer to 45 degrees orvariations thereof, such as 44 and 46 degrees, 40-50 degrees, or anyvariation which may cause minimal distortion of the rendered scene. Asused herein, a horizontal perspective may apply to a single image or astereoscopic image. When used with respect to a stereoscopic image(e.g., presenting a stereoscopic image according to a horizontalperspective), each image of the stereoscopic image may be presentedaccording to the horizontal perspective, but with differing singleviewpoints.

FIGS. 1 and 2—Exemplary Systems

FIGS. 1 and 2 illustrate exemplary systems which are configured toperform various embodiments described below.

In the embodiment of FIG. 1, computer system 100 may include chassis110, display 150A and display 150B (which may collectively be referredto as display 150 or “at least one display” 150), keyboard 120, mouse125, stylus 130, and glasses 140. In one embodiment, at least one of thedisplays 150A and 150B is a stereoscopic display. For example, in oneembodiment, both of the displays 150A and 150B are stereoscopicdisplays.

The chassis 110 may include various computer components such asprocessors, memory mediums (e.g., RAM, ROM, hard drives, etc.), graphicscircuitry, audio circuitry, and other circuitry for performing computertasks, such as those described herein. The at least one memory mediummay store one or more computer programs or software components accordingto various embodiments of the present invention. For example, the memorymedium may store one or more graphics engines which are executable toperform the methods described herein. The memory medium may also storedata (e.g., a computer model) representing a virtual space, which may beused for projecting a 3D scene of the virtual space via the display(s)150. Additionally, the memory medium may store operating systemsoftware, as well as other software for operation of the computersystem. Various embodiments further include receiving or storinginstructions and/or data implemented in accordance with the foregoingdescription upon a carrier medium.

As indicated above, the computer system 100 may be configured to displaya three dimensional (3D) scene (e.g., via stereoscopic images) using thedisplay 150A and/or the display 150B. The computer system 100 may alsobe configured to display a “view” of the 3D scene using the display150A, the display 150B, and/or another display, as described in moredetail below. The “view” of the 3D scene may refer to displaying aportion of the 3D scene from a viewpoint within the 3D scene. Aviewpoint within the 3D scene may be referred to as a “virtualviewpoint”. The view may be stereoscopic, e.g., may be displayed on astereoscopic display. Alternatively, the view may be monoscopic (notstereoscopic), and may be displayed on either a monoscopic display or astereoscopic display.

It should be noted that the embodiment of FIG. 1 is exemplary only, andother numbers of displays are envisioned. For example, the computersystem 100 may include only a single display or more than two displays,or the displays may be arranged in different manners than shown. In thisparticular embodiment, the display 150A is configured as a verticaldisplay (which is perpendicular to a user's line of sight) and thedisplay 150B is configured as a horizontal display (which is parallel oroblique to a user's line of sight). The vertical display 150A may beused (e.g., via instructions sent by a graphics engine executing in thechassis 110) to provide images which are presented according to avertical (or central) perspective and the display 150B may be used(e.g., via instructions sent by a graphics engine executing in thechassis 110) to provide images which are presented according to ahorizontal perspective. Descriptions of horizontal and verticalperspectives are provided in more detail below. Additionally, while thedisplays 150 are shown as flat panel displays, they may be any type ofsystem which is capable of displaying images, e.g., projection systems.

Either or both of the displays 150A and 150B may present (display)stereoscopic images for viewing by the user. By presenting stereoscopicimages, the display(s) 150 may present a 3D scene for the user. This 3Dscene may be referred to as an illusion since the actual provided imagesare 2D, but the scene is conveyed in 3D via the user's interpretation ofthe provided images. In order to properly view the stereoscopic images(one for each eye), the user may wear the glasses 140. The glasses 140may be anaglyph glasses, polarized glasses, shuttering glasses,lenticular glasses, etc. Using anaglyph glasses, images for a first eyeare presented according to a first color (and the corresponding lens hasa corresponding color filter) and images for a second eye are projectedaccording to a second color (and the corresponding lens has acorresponding color filter). With polarized glasses, images arepresented for each eye using orthogonal polarizations, and each lens hasthe corresponding orthogonal polarization for receiving thecorresponding image. With shuttering glasses, each lens is synchronizedto alternations of left and right eye images provided by the display(s)150. The display may provide both polarizations simultaneously or in analternating manner (e.g., sequentially), as desired. Thus, the left eyeis allowed to only see left eye images during the left eye image displaytime and the right eye is allowed to only see right eye images duringthe right eye image display time. With lenticular glasses, images formon cyclindrical lens elements or a two dimensional array of lenselements. The stereoscopic image may be provided via optical methods,where left and right eye images are provided only to the correspondingeyes using optical means such as prisms, mirror, lens, and the like.Large convex or concave lenses can also be used to receive twoseparately projected images to the user.

In one embodiment, the glasses 140 may be used as a position inputdevice to track the eyepoint of a user viewing a 3D scene presented bythe system 100. For example, the glasses 140 may provide informationthat is usable to determine the position of the eyepoint(s) of the user,e.g., via triangulation. The position input device can include aninfrared detection system to detect the position the viewer's head toallow the viewer freedom of head movement or use a light sensitivedetection system. Other embodiments of the input device can be thetriangulation method of detecting the viewer eyepoint location, such asa camera (e.g., a CCD camera) providing position data suitable for thehead tracking objectives of the invention. The input device can bemanually operated by the viewer, such as a keyboard, mouse, trackball,joystick, or the like, to indicate the correct display of the horizontalperspective display images. However, any method for tracking theposition of the user's head or eyepoint is envisioned. Accordingly, the3D scene may be rendered such that user can view the 3D scene withminimal distortions (e.g., since it is based on the eyepoint of theuser). Thus, the 3D scene may be particularly rendered for the eyepointof the user, using the position input device. In some embodiments, eacheyepoint may be determined separately, or a single eyepoint may bedetermined and an offset may be used to determine the other eyepoint.

The relationship among the position/orientation of the display(s) 150and the eye(s) position of the user may be used to map a portion of thevirtual space to the physical space of the system 100. In essence, thephysical space and components used are to be mapped to the virtual modelin order to accurately render a 3D scene of the virtual space. Examplesfor implementing such a system are described in theincorporated-by-reference U.S. patent application Ser. No. 11/098,681entitled “Horizontal Perspective Display” (U.S. Patent Publication No.US 2005/0219694).

One or more of the user input devices (e.g., the keyboard 120, the mouse125, the stylus 130, etc.) may be used to interact with the presented 3Dscene. For example, the user input device 130 (shown as a stylus) orsimply the user's hands may be used to directly interact with virtualobjects of the 3D scene (via the viewed projected objects). However,this direct interaction may only be possible with “open space” portionsof the 3D scene. Thus, at least a portion of the 3D scene may bepresented in this “open space”, which is in front of or otherwiseoutside of (e.g., behind) the at least one display. Thus, at least aportion of the 3D scene may appear as a hologram above the surface ofthe display 150. For example, when the horizontal display 150B is used,the 3D scene may be seen as hovering above the horizontal display. Itshould be noted however, that a portion of the 3D scene may also bepresented as appearing behind the display surface, which is not in “openspace”. Thus, “open space” refers to a space which the user is able tofreely move and interact with (e.g., where the user is able to place hishands in the space) rather than a space the user cannot freely move andinteract with (e.g., where the user is not able to place his hands inthe space, such as below the display surface). This “open space” may bereferred to as a “hands-on volume” as opposed to an “inner-volume”,which may be under the surface of the display(s). Thus, the user caninteract with virtual objects in the open space because they areproximate to the user's own physical space. The inner volume is locatedbehind the viewing surface and presented objects appear inside thephysically viewing device. Thus, objects of the 3D scene presentedwithin the inner volume do not share the same physical space with theuser and the objects therefore cannot be directly, physicallymanipulated by hands or hand-held tools. That is, they may bemanipulated indirectly, e.g., via a computer mouse or a joystick.

In some embodiments, this open space interaction may be achieved byhaving a 1:1 correspondence between the virtual objects (e.g., in thevirtual space) and projected objects (e.g., in the physical space).Thus, an accurate and tangible physical interaction is provided byallowing a user to touch and manipulate projected objects with his handsor hand held tools, such as the stylus 130. This 1:1 correspondence ofthe virtual elements and their physical real-world equivalents isdescribed in more detail in U.S. Patent Publication No. 2005/0264858,which was incorporated by reference in its entirety above. This 1:1correspondence is a new computing concept that may allow the user tophysically and directly access and interact with projected objects ofthe 3D scene. This new concept requires the creation of a commonphysical reference plane, as well as, the formula for deriving itsunique x, y, z spatial coordinates, thereby correlating the physicalcoordinate environment to the virtual coordinate environment.Additionally, the 1:1 correspondence allows the user's movement ofvirtual objects or other interaction (e.g., via the stylus 130) to bethe same in physical space and in presented space. However, otherembodiments are envisioned where there is a ratio between the distanceof the user's physical movement and the corresponding movement in thepresented 3D scene (e.g., of the presented object or virtual stylus).

As described below, the user may be able to specify or otherwisemanipulate a virtual viewpoint within the 3D scene presented by thedisplay(s) 150. A view of the 3D scene may be presented based on thevirtual viewpoint, either by one or more of the display(s) 150 oranother display, as desired. This view of the 3D scene may bestereoscopic or monoscopic, as desired. More details regarding the viewof the 3D scene are provided below.

The 3D scene generator stored and executed in the chassis 110 may beconfigured to dynamically change the displayed images provided by thedisplay(s) 150. More particularly, the 3D scene generator may update thedisplayed 3D scene based on changes in the user's eyepoint,manipulations via the user input devices, etc. Such changes may beperformed dynamically, at run-time. The 3D scene generator may also keeptrack of peripheral devices (e.g., the stylus 130 or the glasses 140) toensure synchronization between the peripheral device and the displayedimage. The system can further include a calibration unit to ensure theproper mapping of the peripheral device to the display images and propermapping between the projected images and the virtual images stored inthe memory of the chassis 110.

In further embodiments, the system 100 (e.g., the display(s) 150) canfurther comprise an image enlargement/reduction input device, an imagerotation input device, and/or an image movement device to allow theviewer to adjust the view of the projection images.

Thus, the system 100 may present a 3D scene which the user can interactwith in real time. The system may comprise real time electronicdisplay(s) 150 that can present or convey perspective images in the openspace and a peripheral device 130 that may allow the user to interactwith the 3D scene with hand controlled or hand-held tools. The system100 may also include means to manipulate the displayed image such asmagnification, zoom, rotation, movement, and even display a new image.

Further, while the system 100 is shown as including horizontal display150B since it simulates the user's visual experience with the horizontalground, any viewing surface could offer similar 3D illusion experience.For example, the 3D scene can appear to be hanging from a ceiling byprojecting the horizontal perspective images onto a ceiling surface, orappear to be floating from a wall by projecting horizontal perspectiveimages onto a vertical wall surface. Moreover, any variation in displayorientation and perspective (or any other configuration of the system100) are contemplated.

FIG. 2 illustrates another embodiment of the system 100, shown as 200Aand 200B. In this embodiment, the system may be a foldable and/orportable system (e.g., similar to a laptop or tablet computer) where theuser may have the system 200 open (as shown in 200A) or closed (as shownin 200B). In this embodiment, the horizontal display and verticaldisplay may be blended by a blending display. Thus, the display of thesystem 200 may be thought of as a plurality of combined displays, or asingle display which is able to project horizontally and/or vertically,as desired.

Exemplary Systems

Embodiments of the present invention may augment the current state ofreal-time computer-generated 3D computer graphics and tactilecomputer-human interfaces with real time interaction. More specifically,these new embodiments may enable real-time computer-generated 3Dsimulations to coexist in physical space and time with the userinteracting with the projected objects. This unique ability may beuseful in many industries including, but not limited to, electronics,computers, biometrics, medical, education, games, movies, science,legal, financial, communication, law enforcement, national security,military, print media, television, advertising, trade show, datavisualization, computer-generated reality, animation, CAD/CAE/CAM,productivity software, operating systems, and more.

FIGS. 3A and 3B—Horizontal and Vertical Perspective

FIG. 3A illustrates an exemplary diagram of a horizontal perspectiveprojection and FIG. 3B illustrates an exemplary diagram of a verticalperspective projection.

In the horizontal perspective of FIG. 3A, the projected image is not onthe plane of vision—instead, it is on a plane angled to the plane ofvision. Typically, the image would be on the ground level surface. Thismeans the image will be physically in the third dimension relative tothe plane of vision. As indicated above, it may be desirable orimportant that the image is viewed from the correct eyepoint, otherwisethe 3D scene may not represent a physical truism.

In FIG. 3A, the object was drawn by the artist closing one eye, andviewing along a line of sight 373 45° to the horizontal display plane374. The resulting image, when viewed horizontally at the eyepoint, (inthis case, for a single image at 45° and through one eye) looks the sameas the original image. In FIG. 3B, the object in the 3D scene (threeblocks stacked slightly above each other) was drawn by the artistclosing one eye, and viewing along a line of sight 371 perpendicular tothe vertical display plane 372. The resulting image, when viewedvertically, straight on, and through one eye, looks the same as theoriginal image.

As can be seen, one major difference between vertical (e.g., central)perspective showing in FIG. 3B and horizontal perspective in FIG. 3A isthe location of the display plane (374 and 372) with respect to theprojected 3D image. In the horizontal perspective of FIG. 3A, thedisplay plane can be adjusted up and down, and therefore the projectedimage can be conveyed in the open air above the display plane, e.g., auser can touch (or more likely pass through) the illusion, or it can bedisplayed under the display plane, e.g., a user cannot touch theillusion because the display plane physically blocks the hand. This isthe nature of horizontal perspective, and as long as the renderingviewpoint and the user's eyepoint are at the same place, the illusion ispresent. In contrast, for the single eye vertical (e.g., central)perspective of FIG. 3B, the 3D illusion is likely to be only inside thedisplay plane, meaning one cannot touch it. However, using stereoscopicimages, both perspectives can convey the 3D scene in “open space”.

FIGS. 4A and 4B—Display Adjustment for Horizontal Perspective

The display(s) 150 may be made of many physical layers, individually andtogether having thickness or depth. To illustrate this, FIGS. 4A and 4Billustrate a conceptual side-view of a typical LCD display 450 (anembodiment of one or more of the displays 150). FIGS. 4A and 4B alsoillustrate how the projected 3D scene can include a portion in openspace and another portion in the inner volume.

The top layer of the display 450 is the physical “view surface” 452, andthe imaging layer (liquid crystal layer), where images are made, is thephysical “image layer” 453. The view surface 452 and the image layer 453are separate physical layers located at different depths or zcoordinates along the viewing device's z axis. To display an image, theLCD's polarized light is provided from the image layer 453 through theview surface 452 (which may be glass).

In the example shown in FIGS. 4A and 4B, the same blocks from 3A and 3Bare shown with a horizontal perspective. As shown, the middle block inFIG. 4A does not correctly appear on the view surface 452. In FIG. 4A,the imaging layer, i.e. where the image is made, is located behind theview surface 452. Therefore, the bottom of the middle block isincorrectly positioned behind or underneath the view surface 452.

FIG. 4B illustrates an example of the proper location of the threeblocks on the display 450. That is, the bottom of the middle block isdisplayed correctly on the view surface 452 and not on the image layer453. To make this adjustment, the z coordinates of the view surface 452and image layer 453 are used by the 3D scene generator to correctlyrender the image. Thus, the unique task of correctly rendering an openspace image on the view surface 452 versus the image layer 453 may becritical in accurately mapping the 3D scene objects to the physicalprojected space.

Thus, the display's view surface 452 is the correct physical location todemarcate the division between open space and inner space and henceimage rendering must use this view surface thickness as an offset whenintending to render scenes where the object is to be fully conveyed inopen space. Therefore, the top of the display's view surface 452 is thecommon physical reference plane. However, only a subset of the viewsurface 452 can be the reference plane because the entire view surfacemay be larger than the total image area.

Many viewing devices enable the end user to adjust the size of the imagearea within the viewing region of the viewing devices by adjustingcertain x and y values. But all three, x, y, z, coordinates areessential to determine the location and size of the common physicalreference plane. The formula for this is: The image layer 453 is given az coordinate of 0. The view surface is the distance along the z axisfrom the image layer 453. The reference plane's z coordinate is equal tothe view surface 452, i.e. its distance from the image layer 453. The xand y coordinates, or size of the reference plane, can be determined bydisplaying a complete image on the viewing device and measuring thelength of its x and y axis.

The concept of the common physical reference plane is not common.Therefore, display manufactures may not supply its coordinates. Thus a“reference plane calibration” procedure might need to be performed toestablish the reference plane coordinates for a given display surface.This calibration procedure may provide the user with a number oforchestrated images with which he interacts. The user's response tothese images provides feedback to the 3D scene generator such that itcan identify the correct size and location of the reference plane. Inone embodiment, when the end user is satisfied and completes theprocedure the coordinates are saved in the end user's personal profile.With some displays, the distance between the view surface 452 and imagelayer 453 is quite small. But no matter how small or large the distance,it is critical that all Reference Plane x, y, and z coordinates aredetermined as close as technically possible within certain tolerance,e.g., optimally less than a quarter inch.

After the mapping of the “computer-generated” horizontal perspectiveprojection display plane to the “physical” reference plane x, y, zcoordinates, the two elements are essentially coincident in space; thatis, the computer-generated horizontal plane now shares the real-world orphysical x, y, z coordinates of the physical reference plane.

FIGS. 5A and 5B—Exemplary Mono and Stereo View Volumes in HorizontalPerspective

FIGS. 3A and 3B illustrate “view volumes” of the horizontal and verticalperspectives, respectively, for a single eye. FIG. 5A illustrates a moredetailed single eye view volume in a horizontal perspective, and FIG. 5Billustrates the view volumes of a stereoscopic image.

Mathematically, the computer-generated x, y, z coordinates of theviewpoint (e.g., corresponding to a user's eyepoint) form the vertex ofan infinite pyramid, whose sides pass through the x, y, z coordinates ofthe reference/horizontal plane. FIG. 5A illustrates this infinitepyramid, which begins at the viewpoint 551 and extends through the farclip plane (not shown). There are new planes within the pyramid that runparallel to the reference/horizontal plane 556, which, together with thesides of the pyramid, define two new volumes. These unique volumes arecalled open space volume 553 and the inner volume 554, which weredescribed previously. As shown, the open space volume 553 may existwithin the pyramid between and inclusive of the comfort plane 555 andthe reference/horizontal plane 556. As indicated above, in oneembodiment, a user cannot directly interact with 3D objects locatedwithin the inner volume 554 via his hand or hand held tools (such as thestylus 130), but they can interact in the traditional sense with acomputer mouse, joystick, or other similar computer peripheral. Theplane 556 along with the bottom plane 552, are two of the planes withinthe pyramid that define the inner volume 554. Note that while the bottomplane 552 is farthest away from the viewpoint 551, it is not to bemistaken for the far clip plane.

FIG. 5A also illustrates a plane 555, called the comfort plane. Thecomfort plane 555 is one of six planes that define the open space volume553, and of these planes it is closest to the viewpoint 551 and parallelto the reference plane 556. The comfort plane (or near plane) 555 isappropriately named because its location within the pyramid determinesthe user's personal comfort, e.g., how his eyes, head, body, etc. aresituated while viewing and interacting with simulations. The user canadjust the location of the comfort plane 555 based on his personalvisual comfort through a “comfort plane adjustment” procedure, where theuser can adjust the position or closeness of the plane 555. Thisprocedure may provide the user with various 3D scenes within the openspace volume 553 and may enable him to adjust the location of thecomfort plane 555 within the pyramid relative to the reference plane556. When the user is satisfied and completes the procedure, thelocation of the comfort plane 555 may be saved in the user's personalprofiles. Other planes, such as the bottom plane may be adjustedsimilarly.

FIG. 5B illustrates the provision of a stereoscopic image to two singleviewpoints (corresponding to two eyes) viewing the 3D scene of the polarbear. As shown, viewpoint 662 may correspond to a user's right eyepointand viewpoint 664 may correspond to a user's left eyepoint. By renderingand presenting a stereoscopic image according to these singleviewpoints, a 3D scene of the polar bear may be provided to the user,e.g., using the glasses 140 as described above.

FIG. 6—Presenting a View Based on a Virtual Viewpoint within a 3D Scene

FIG. 6 illustrates a method for presenting a view based on a virtualviewpoint within a 3D scene. The method shown in FIG. 6 may be used inconjunction with any of the computer systems or devices shown in theabove Figures, among other devices. In various embodiments, some of themethod elements shown may be performed concurrently, in a differentorder than shown, or may be omitted. Additional method elements may alsobe performed as desired. As shown, this method may operate as follows.

In 602, a 3D scene may be presented by at least one display (e.g., thedisplay(s) 150). More particularly, one or more stereoscopic images ofthe 3D scene may be presented by the display(s). The 3D scene may bepresented according to a first viewpoint. For example, the firstviewpoint may be based on an eyepoint of a user viewing the 3D scene. Inone embodiment, the method may include determining the first viewpoint,e.g., by determining the eyepoint of the user viewing the 3D scene. Themethod may determine the eyepoint of the user using various techniques,such as a position input device (e.g., glasses which provide eyepointposition information), triangulation, head/eye tracking, etc.Accordingly, the 3D scene may be rendered such that the user can viewthe 3D scene with minimal distortions (e.g., since it is based on theeyepoint of the user). More specifically, when the 3D scene is based onthe user's eyepoint, the 3D scene is rendered based on the perspectiveas would be seen by the viewer. This rendering avoids much of thedistortion that would be conveyed if the viewpoint of the scene did notmatch the eyepoint of the viewer. In other words, a displayed objectretains the correct perspective as perceived by the viewer as long asthe viewer eyepoint and 3D scene viewpoint remain in correspondence

As indicated above, the 3D scene may be presented by a single display ora plurality of displays. In one embodiment, the 3D scene may bepresented by a vertical display and a horizontal display. For example,the vertical display may present a first stereoscopic image, e.g.,according to a vertical perspective, and the horizontal display maypresent a second stereoscopic image, e.g., according to a horizontalperspective. These two stereoscopic images may form or convey the 3Dscene to the user. In further embodiments, the two displays may bejoined by a curvilinear or blending display, which may also present astereoscopic image. The stereoscopic image of the blending display mayoperate to blend the stereoscopic images of the vertical and horizontaldisplays. Other numbers and types of displays are contemplated forpresenting the 3D scene.

At least a portion of the 3D scene may be presented in “open space” infront of or otherwise outside of the at least one display. Thus, atleast a portion of the 3D scene may appear as a hologram above thedisplay surface. For example, when a horizontal display is used, the 3Dscene may be seen as hovering above the horizontal display. It should benoted however, that a portion of the 3D scene may also be presented asappearing behind the display surface, which is not in “open space”.Thus, “open space” refers to a space which the user is able to freelymove and interact (e.g., where the user is able to place his hands inthe space) rather than a space the user cannot freely move nor interact(e.g., where the user is not able to place his hands in the space, suchas below the display surface). This “open space” may be referred to as a“hands-on volume” as opposed to an “inner-volume”, which may be underthe surface of the display(s). Thus, the user can directly interact withobjects (displayed virtual objects) in the open space because theyco-inhabit the physical space proximate to the user. The inner volume islocated behind the viewing surface, and portions of the 3D scene withinthis inner volume appear “inside” the physical viewing device. Thus,objects of the 3D scene presented within the inner volume do not sharethe same physical space with the user, and the objects therefore cannotbe directly, physically manipulated by hands or hand-held tools. Thatis, objects displayed within the inner volume may be manipulatedindirectly, e.g., via a computer mouse or a joystick.

In 604, a virtual viewpoint within the 3D scene may be determined, wherethe virtual viewpoint is different than the first viewpoint. Forexample, the virtual viewpoint within the 3D scene may be selected orotherwise specified by a user, e.g., the user viewing the 3D scene. Inone embodiment, the user may manipulate a handheld device, such as astylus, to specify the virtual viewpoint within the 3D scene. Forexample, the user may use the handheld device to specify a correspondingvirtual viewpoint within the 3D scene by positioning the handheld devicein the open space of the presented 3D scene. Thus, the user input may be“open space user input” to specify the virtual viewpoint.

The virtual viewpoint may be the viewpoint of a virtual object displayedin the 3D scene. For example, a virtual camera object may be one of manyvirtual objects rendered and conveyed within the 3D scene, and the usermay be able to manipulate the virtual camera object (e.g., an objectthat resembles a camera) in the 3D scene (e.g., using a stylus in openspace as indicated above) by positioning and/or orienting the stylus tospecify the virtual viewpoint of the virtual camera object within the 3Dscene. Alternatively, or additionally, the virtual object may be amagnifying glass (e.g., an object resembling a magnifying glass) orother type of tool or displayed virtual object. In further embodiments,the virtual object may be an avatar or entity in the scene (e.g., aperson or bird flying through the 3D scene) and the virtual viewpointmay correspond to an eyepoint of the virtual entity within the 3D scene.

In addition to determining the virtual viewpoint, a field of view of thevirtual viewpoint may be determined. For example, the field of view maybe a wide angle field of view (e.g., corresponding to a wide angle lensof a virtual camera object) or a telescopic field of view (e.g.,corresponding to a telescopic lens of a virtual camera object).Additionally, a magnification for the virtual viewpoint may bedetermined. The field of view and/or magnification may be determinedbased solely on the virtual viewpoint, or may be determined based onother factors, e.g., user input. For example, the user may specify atelescopic lens or a wide angle lens. The user may also be able toselect or specify a specific magnification (e.g., of the telescopiclens). Thus, in some embodiments, the field of view and/or magnificationmay be determined based on user input. Further the view volume of thevirtual viewpoint may be determined, e.g., automatically or based onuser input. More specifically, the near and far clip planes of thevirtual viewpoint volume may be determined, and the view of the 3D scenemay be presented based on this determined view volume.

In some embodiments, the 3D scene may indicate the virtual viewpoint.For example, the displayed 3D scene may illustrate the frustum of theview of the 3D scene within the 3D scene. In one particular example, thevirtual camera object displayed in the 3D scene may also indicate thefrustum of view of the virtual object within the 3D scene.

In 606, a view of the 3D scene may be presented on the at least onedisplay according to the virtual viewpoint. Said another way, the viewof the 3D scene may be rendered from the perspective of the virtualviewpoint specified above. Note that the view of the 3D scene may bepresented concurrently with the presentation of the 3D scene. Where thefield of view and/or magnification is determined above, the presentationof the view of the 3D scene may be based on the determined (orspecified) field of view and/or magnification. For example, if amagnification is specified, the view of the 3D scene may be magnified atthat corresponding magnification. Thus, the view of the 3D scene may bemagnified compared to a corresponding portion of the 3D scene.

The presentation of the view of the 3D scene may also be based on thefirst viewpoint (that corresponds to the eyepoint of the user). Forexample, the view of the 3D scene may be presented such that it is easyto see based on the first viewpoint (e.g., where it is perpendicular tothe line of sight of the user viewing the 3D scene).

The view of the 3D scene may be presented in any number of ways. Forexample, the view of the 3D scene may be monoscopic or stereoscopic, asdesired. Additionally, the view of the 3D scene may be presented on thesame display as the 3D scene or on different displays. For example, theview of the 3D scene may be presented within the 3D scene. In otherwords, the view of the 3D scene may be presented on the same display, oron the same plurality of displays, which are used to present the 3Dscene itself. Alternatively, or additionally, the view of the 3D scenemay be provided on a separate display, e.g., which is dedicated to theview of the 3D scene. Thus, in one embodiment, the at least one display(which is used to present the 3D scene and the view of the 3D scene) mayinclude a first one or more displays and a second one or more displays,and the 3D scene may be provided via the first one or more displays andthe view of the 3D scene may be presented via the second one or moredisplays.

Additionally, the view of the 3D scene may be presented at a positionand orientation which differs from that of the display providing theview of the 3D scene. For example, the display may have a horizontalposition and the view of the 3D scene may be presented on a virtualdisplay which faces (or is perpendicular to) the eyepoint of the userviewing the 3D scene. The view of the 3D scene (or virtual display) maybe positioned in open space of the 3D scene, e.g., in open spacerelative to the display providing the view of the 3D scene.

In 608, the 3D scene and/or the view of the 3D scene may be updatedbased on a change in the first viewpoint and/or a change in the virtualviewpoint.

For example, the method may further include moving the virtualviewpoint, e.g., in response to user input. The movement of the virtualviewpoint may be performed similarly to the specification of the virtualviewpoint described above. For example, the user may select the virtualobject specifying the virtual viewpoint (e.g., the virtual cameraobject) and may change the position and/or orientation of the virtualobject in the 3D scene to move the virtual viewpoint from an originalposition and/or orientation to a second position and/or orientation.

The movement of the virtual viewpoint may be discrete (e.g., immediatelyfrom the original to the second) or may be a continuous movement. Ineither case, the view of the 3D scene may be updated according to themovement, e.g., discretely or smoothly. In the continuous movement, the3D scene may show the continuous change in viewpoint from the originalviewpoint to the finally specified viewpoint. Thus, the determination ofa virtual viewpoint (504) and the presentation of the view of the 3Dscene (506) may be performed a plurality of times throughoutpresentation of the 3D scene, and in particular, throughout movement orchange of the virtual viewpoint.

The presentation of the view of the 3D scene may be performed at a framerate or quality which is comparable to the presentation of the 3D scene.Additionally, or alternatively, the view of the 3D scene may be providedat a frame rate or quality which is above a certain threshold. Thisquality or speed may ensure that the view of the 3D scene is provided ina smooth manner, e.g., at a rate where the user does not notice movementartifacts in the view of the 3D scene, such as choppiness or glitching.This quality of presentation may apply when the virtual viewpoint isstatic (e.g., when objects are moving within the view of the 3D scene)or when the virtual viewpoint is being moved (e.g., the movement of thevirtual viewpoint described above).

Additionally, the 3D scene may be updated based on changes of the firstviewpoint (e.g., corresponding to changes of the eyepoint of a user).For example, the user may move his head, thereby changing the eyepointlocation. Accordingly, a next viewpoint (which corresponds to the user'snew eyepoint) may be determined after displaying the 3D scene and theview of the 3D scene. Based on this next viewpoint, the 3D scene may beupdated and an updated stereoscopic image of the 3D scene may beprovided by the display(s). Where the view of the 3D scene is also basedon the viewpoint corresponding to the eyepoint of the user, the view ofthe 3D scene may be correspondingly updated as well.

FIG. 7—Exemplary 3D Scene and View of the 3D Scene

FIG. 7 illustrates an exemplary 3D scene and view of the 3D sceneaccording to the method of FIG. 6. This Figure is exemplary only andother 3D scenes, views of the 3D scene (including size, orientation,position, etc.), and mechanisms of presenting the 3D scene (e.g.,numbers and use of display devices) are envisioned.

In the example of FIG. 7, a single horizontal display 150B is used. Thehorizontal display 150B may provide a stereoscopic image of the anatomyof an arm (in this case the bones of the arm). Within the 3D scene, avirtual object (in this case, a virtual camera object) 720 may bedisplayed. The user may be able to manipulate this virtual object viaany number of mechanisms, such as the stylus 130, described above. Forexample, the user may select the virtual object 720 for changing itsposition or orientation. In response, the view of the 3D scene from theviewpoint of the virtual object 720 may be presented in the virtualdisplay 750. In the embodiment shown, images of the virtual display 750may not be stereoscopic (e.g., the same image of the virtual display 750may be provided to both of the user's eyes). Additionally, the virtualdisplay 750 has a different position and orientation than the realdisplay 150B.

Thus, FIG. 7 illustrates an exemplary system where the user maymanipulate the virtual object 720 to specify a virtual viewpoint. Asdescribed in the method of FIG. 6, this virtual viewpoint may be used torender a view of the 3D scene on a virtual display 750 projected abovethe surface of the horizontal display 150B.

FIGS. 8A-16B—Exemplary Illustrations

FIGS. 8A-16B illustrate various virtual spaces and corresponding 3Dscenes of the virtual spaces.

FIG. 8A illustrates a virtual representation, and FIG. 8B illustratespresentation of a 3D scene of the virtual space using the display 150A.Similar to descriptions above, the display 150A may be a stereo 3Ddisplay conveying a stereoscopic (left and right) image for stereoviewing.

In the virtual representation of FIG. 8A, various virtual objects of avirtual space are shown (objects 1-7). These seven objects are each inan x-y-z position in the virtual space as modeled within a computer.Additionally, the viewpoint 820 (including single viewpoints 820A and820B) corresponds to the user's eyepoint in the physical space. Further,the drawing plane 810 corresponds to the display 150A in the physicalspace. In the FIG. 8A, the viewpoint 820 is represented by a device withtwo virtual eyes having viewpoints 820A and 820B, but in practice asingle viewpoint may be used, but a stereo pair may be generated using aplus offset and/or a minus offset, where the plus and minus offsetapproximate the inter pupil distance of the user's eyepoint 870.

As shown, two frustums 830A and 830B are generated of a view from theviewpoint 820 (corresponding to single viewpoints 820A and 820B, whichcorrespond to the eyepoint 870). In this specific example, the nearplane of the frustums 830A and 830B correspond to the viewpoints 820Aand 820B, but may not in other examples. As shown, objects 4, 5, and 7are excluded from these frustums 830A and 830B and are therefore notpresent in the projection of FIG. 8B. The display 150A presents an imagefor each frustum of 830A and 830B which are presented to the user inorder to present the 3D scene 850 of FIG. 8B. Thus, as shown in FIG. 8B,the virtual objects 1, 2, 3, and 6 are presented to the user viapresentation of stereoscopic image(s) by the display 150A. However, itshould be noted that only portions of objects 1, 2, 3, and 6 arepresented where the whole of objects 1, 2, 3, and 6 are not present inthe frustums 830A and 830B.

FIG. 9A illustrates a virtual representation 900, e.g., formagnification, and FIG. 9B illustrates a presentation of the 3D scene950 of FIG. 9A (using the display 150A) as well as a 3D view of thescene of FIG. 9A (using a separate display 960, which may be similar todisplay 150A).

As shown within the virtual representation 900, the same objects as FIG.8A are present. However, in this Figure a new viewpoint (a virtualviewpoint) 925 is shown. In this particular embodiment, the virtualviewpoint 925 (which is a stereoscopic viewpoint in this embodiment) isfrom the point of view of a virtual object 925. Objects 1 and 2 arepresent with the view volume 927 of the viewpoint 925. In the embodimentof FIG. 9B, within the 3D scene 950, the same objects as were previouslyshown in FIG. 8B are still presented to the user via stereoscopicimage(s). In addition, the virtual object 925 and the view volume 927 isshown in the 3D scene 950. Further, on the display 960 a view of the 3Dscene 980 from the point of view of the viewpoint 925 is shown for theuser. As shown, the user views object 1 and object 2 of the view of the3D scene from the position/orientation of the virtual object 925. Thus,the user may both view the entire 3D scene (which is just a portion ofthe virtual space 900) including the virtual object 925 and its viewvolume via display 150A and the view of the 3D scene from the virtualobject 925 via display 960. Moreover, the 3D scene may be rendered forthe eyepoint of the user from the viewpoint 920 and the view of the 3Dscene may be rendered for the eyepoint of the user from the viewpoint925 concurrently. In some embodiments, the view volume (from 920 to 910)from the viewpoint 920 may be considered a “wide angle view” while theview from viewpoint 925 may be considered a “narrow view”. Thus, the“narrow view” may be a subsection of the “wide angle view”.

The user may be able to manipulate the location and orientation of theviewpoint 925, as described above. Additionally, the user may be able tomanipulate the view volume 927. By manipulating these parameters (amongother possible parameters), the user may be able to magnify portions ofthe 3D scene for presentation by the display 960, thereby achieving a 3Dzoom. Thus, instead of the conventional magnification or loupe tool witha two dimensional select region boundary, a three dimensional volume maybe specified or selected. The 3D volume 927 may be any number of shapes,e.g., which may be controlled or specified by a user using an inputdevice, such as a mouse, keyboard, or 3D input device. In oneembodiment, the volume has a near boundary and a far boundary, whereeach of these two boundaries may be set at any position so the distancefrom the near and far boundaries may be close to each other or far apart(or anything in between). Furthermore, as indicated above, the positionand orientation of the viewpoint 925 may be modified by the user.

However, in further embodiments, the user may simply specify the volume927, and the viewpoint 925 may be determined automatically (e.g., andmay not be shown in the 3D scene 950). Though there may be a positionand/or orientation relationship between the viewpoint 925 and itscorresponding view volume 927, they may be established independent ofeach other. For example, the viewpoint position/orientation relative tothe view volume may be set by the user or may be a default. Furthermore,the view volume 927 itself may be set by the user or may be a default.The control of the movement of the selected volume 927 or the viewpoint925 may cause re-rendering of one or more of the 3D scene(s) 950 and theview of the 3D scene 980 in a transparent and smooth manner.

As indicated above, the display 960 may be a distinct display or may bea virtual display within the 3D scene 950 presented by the display 150A.Additionally, the view of the 3D scene 980 may be stereoscopic ormonoscopic, as desired. Further, the projection of the view of the 3Dscene 980 may be rendered dependent or independent of theposition/orientation of the display 960. Additionally, this renderingmay be set by the user or may be a default. Similar remarks apply to thedisplay 150A.

FIG. 10A illustrates a virtual representation 1000, and FIG. 10Billustrates a presentation of the 3D scene 1050 of FIG. 10A (using thedisplay 150B) as well as a 3D view of the scene of FIG. 10A (using thesame display 150B).

The descriptions of FIGS. 9A and 9B essentially apply to FIGS. 10A and10B; however, as shown, instead of two distinct vertical displays, asingle horizontal display 150B is used. Using this display 150B, twodifferent 3D projections are provided, a first (1050) from the viewpoint1020 using a portion (1055) of the display 150B and a second (1080) fromthe view point 1025, both rendered for the eyepoint 1070. Thus, thedisplay 150B may provide a stereoscopic horizontal projection usingdisplay portion 1055 for the viewpoint 1020 (which directly correspondsto the eyepoint 1070) and may also provide a stereoscopic horizontalprojection using display portion 1060 for the viewpoint 1025 (which doesnot directly correspond to the eyepoint 1070, but is rendered for thateyepoint). As shown, the 3D scene includes the four objects 1, 2, 3, and6, the virtual object 1025, and the view volume 1027. The view of the 3Dscene 1080 includes the objects 1 and 2 corresponding to the view volume1027.

FIG. 11 illustrates a particular embodiment of the view of the 3D scene1080, provided by the vertical display 150A. In this embodiment, theportion 1160 of the display 150A (similar to portion 1060 on thehorizontal display 150B) may be used to present the view of the 3D scene1080. In this particular embodiment, the user may see the view of the 3Dscene 1080 from a virtual display 1162, which is arranged on a lineperpendicular to the line of sight from the eyepoint 1070, rather thansimply perpendicular to the display 150A. Thus, FIG. 11 illustrates anembodiment where a virtual display 1162 is used to present the view ofthe 3D scene 1080 to the user stereoscopically. Additionally, thevirtual display 1162 has a different position and orientation than theportion 1160 of the screen 150B.

In the embodiment shown, this projected virtual display 1162 ispresented stereoscopically as a left-right image pair, such that whenviewed by the user at the eyepoint 1070 with both eyes, it is perceivedas extending out from the surface of the display 150B. However, thevirtual display 1162 may be presented in a manner such that it appearsbehind the display 150B, in front of the display 150B, and/or at one ofmany angles and orientations. These setting may be dependent upon theuser eyepoint position 1070, user selectable, or at some default andconveyed at one of many projections.

FIG. 12 illustrates an alternate embodiment where the virtual display1162 is as before, but the region volume objects are conveyed inmonoscopic, being a single eye view. The mono conveyance appears to theuser to be at the virtual surface of the virtual display 1162 ratherthan in front of the display, as in FIG. 11.

FIG. 13 illustrates a similar embodiment to FIG. 11, except the verticaldisplay 150A is replaced with the horizontal display 150B (also similarto FIGS. 10A and 10B). Similar to FIG. 11, the portion 1055 may be usedto present the 3D scene 1050, and the portion 1060 may be used topresent the view of the 3D scene 1080. However, similar to theembodiment of FIG. 11, the portion 1060 may be used to present a virtualdisplay 1362 for presenting the view of the 3D scene 1080. Variationssimilar to the virtual display 1162 apply to the virtual display 1362.

FIG. 14A illustrates the virtual representation 1000 with an interactiveelement 1445, which may act as an extension to stylus 130 (shown in FIG.14B and described below), and FIG. 14B illustrates the presentation ofthe 3D scene 1050 corresponding to the virtual representation 1000 ofFIG. 14A (using the display 150B) as well as a 3D view of the scenecorresponding to the view volume 1027 of FIG. 14A (using the samedisplay 150B).

In this embodiment, the user may interact with the virtual object 1025and/or the view volume 1027 via the stylus 130. In the embodiment shown,the physical stylus 130 may be extended in the 3D scene 1050 via theprojected, virtual stylus 1445. Thus, the stylus 130 may have a virtual,displayed extension 1445 which begins from the end of the physicalstylus 130. To specify the viewpoint 1025 and/or the view volume 1027,the user may use the stylus 130 in a position for manipulating theviewpoint 1025 (e.g., for selection and movement) or the view volume1027 (e.g., to specify the boundaries of the view volume 1027).

The hand held tool may be any tracked device, e.g., in terms of positionand orientation. The stylus 130 may be of a variety of shapes and it mayor may not have the virtual stylus 1445, as an extension or otherwise.Additionally, instead of acting as an extension, the virtual stylus 1445may move according to corresponding movements of the stylus 130, but maynot appear to be connected at all. In the embodiment shown, the stylus130 has the virtual stylus 1445 that is appears as an extension to thestylus 130, e.g., via stereoscopic images provided to the eyepoint 1070.

As the user moves the stylus 130, certain corresponding actions mayoccur. In one embodiment, the corresponding virtual stylus 1445 isrepositioned both in the virtual representation and the 3D scene. Inanother embodiment, the viewpoint 1025 and/or the view volume 1027 maybe specified or moved. For example, the view volume 1027 may have itsboundaries adjusted. In another embodiment, the virtual stylus 1445 maynot be rendered and hence may not be imaged nor seen by the user, butthe viewpoint object 1025 may allow the user to have a feedbackmechanism of where the stylus 130 is pointing. The imaged distancebetween the far tip of the stylus 130 or the virtual stylus 1445 and theuser perceived imaged viewpoint object 1025 can be set based ondifferent parameters, such as user or design preferences. The higher theresolution of the position/orientation accuracy of the stylus 130, thecloser the viewpoint object 1025 may be to it. At times it may bedesirable to have the viewpoint object 1025 positioned to be coincidentto the far tip of the stylus 130 or the virtual stylus 1445. That waythe user may accommodate his eyes to the hand held tool tip, whichallows for a truer ability to position the viewpoint object 1025 inrelation to the stereo imaged virtual objects the user may wish toprobe.

It is also possible to have multiple virtual volumes within a 3D scene,e.g., where one view volume is within or includes a second view volume.FIG. 15A illustrates the virtual representation 1500 with view volume1027 from viewpoint 1025 as well as another view volume 1537 fromviewpoint 1535, which has a different position and orientation than viewvolume 1027 and overlaps with view volume 1027. FIG. 15B illustrates aportion of the presentation of the 3D scene 1550 of FIG. 15A (using thedisplay 150B) as well as a 3D view of the scene of FIG. 15A (using thesame display 150B) according to the viewpoint 1535.

As shown, the display 150B may include three portions, 1055 for the 3Dscene 1550, 1060 for the view of the 3D scene 1080 (not shown in FIG.15B), and 1562 for the view of the 3D scene 1085. Since the view volume1537 only includes object 2, the corresponding view of the 3D scene 1085only includes object 2. Similar to above, the view volume may bepresented according to any of the various possibilities alreadyenumerated above.

Different methods for activating a subsequent view volumes or viewpointsinclude keyboard selection, mouse selection, stylus 130, virtual stylus1445, etc. As each viewpoint or view volume is created, the previouslycreated viewpoint or view volume may either remain or may be removed,either by user input or by default, as desired. For those viewpoints orview volumes that remain, their positions within the virtual space maybe stored, e.g., as a bookmark for later revisiting. The creation ofthese viewpoints or view volumes may be specified by placing the stylus130 in a position and having that position be stored, so the stylus 130may then be moved to a resting position or for establishing subsequentviewpoints or view volumes.

FIG. 16A illustrates the virtual representation 1600 with an interactiveelement 1445, which may act as an extension to stylus 130, and FIG. 16Billustrates the presentation of the 3D scene 1650 of FIG. 16A (using thedisplay 150A) as well as a 3D view of the scene of FIG. 16A (using asecond display 960).

In the embodiment shown, the user may manipulate objects behind thedisplay 150A via the stylus 130 and/or virtual stylus 1445, e.g., tomanipulate viewpoint 1635 and/or view volume 1637, which includesobjects 8 and 9. In this embodiment, the display 150A may be asee-through OLED prototype laptop with a 14-inch transparent,see-through color OLED screen, which may be capable of providingstereoscopic images according to the viewpoint 920. With the see-throughdisplay, the stylus 130 may be used in both in front of the display 150Aand behind the display 150A. As stereo imagery is rendered to be seenboth in front of the screen and behind the screen, with the see-throughdisplay 150A, the user may position the stylus 130 (or even the virtualstylus 1445) on either side of the display 150A. Thus, the user mayposition the stylus 130 (e.g., for use as or for manipulating a virtualmagnifier, virtual camera, or probe tool) on either side of the display,and the extension 1445 (when present) may be shown on either side aswell, via rendered stereoscopic images. Thus, in one embodiment, thevirtual stylus 1445 may extend the stylus 130 regardless of whether thestylus 130 is in front of or behind the screen 150A.

In the embodiment shown, the virtual representation, displays, and 3Dscenes may be similar to that of FIGS. 9A and 9B, save for the additionof the two objects 8 and 9 (shown behind the screen 960 and 150A) andthe user manipulation of FIGS. 14A and 14B. All of the variationsdescribed above may also apply to this figure, e.g., with multiplevirtual viewpoints and view volumes, etc.

Further Embodiments

In further embodiments, the virtual viewpoint described above may berecorded and saved for later playback. For example, the viewpoint may bebookmarked Additionally, where the virtual viewpoint corresponds to avirtual object, e.g., a virtual camera, it may be possible to record theview of the 3D scene, e.g., as a movie, from the view point of thevirtual camera. This recording may be stored and played back at a latertime. Further, the virtual object (e.g., the virtual camera) may be ableto illuminate a portion of the 3D scene being viewed by the camera,e.g., similar to a flashlight.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications will become apparent tothose skilled in the art once the above disclosure is fully appreciated.It is intended that the following claims be interpreted to embrace allsuch variations and modifications.

We claim:
 1. A system for presenting a three dimensional (3D) scene,comprising: at least one display; and at least one processor coupled tothe at least one display; wherein the at least one processor isconfigured to: determine a user viewpoint based on tracking of a userposition, wherein the user position comprises position and orientationin physical space; determine a user perspective relative to at least onedisplay surface of the at least one display, wherein the userperspective comprises a mapping between angle and orientation of the atleast one display surface and a render plane to the user viewpoint;render and display the 3D scene within a virtual space based on aprojecting in virtual space to the render plane, wherein the renderplane has a correlation to the position and orientation of the at leastone display, and wherein the correlation is based on the userperspective; determine first and second virtual viewpoints correspondingto first and second positions, angles and orientations in the virtualspace, wherein the first and second virtual viewpoints are based onfirst and second positions, angles and orientations of at least aportion of a hand of the user in the physical space without use ofhand-held tools; and establish first and second fields of view and firstand second view volumes of the 3D scene based on the first and secondvirtual viewpoints.
 2. The system of claim 1, wherein the userperspective is based on a first user eypoint and the user position. 3.The system of claim 2, wherein to determine the user perspective, the atleast one processor is further configured to: determine a first usereyepoint based on user position; and correlate user position to aposition of the at least one display surface, wherein the correlation isrelative to an angle and orientation of the at least one displaysurface.
 4. The system of claim 1, wherein the at least one processor isfurther configured to: store the first and second fields of view and thefirst and second view volumes.
 5. The system of claim 1, wherein thefirst and second positions, angles, and orientations of the at least aportion of the hand of the user are in open space.
 6. The system ofclaim 1, wherein the user position comprises at least one of: a userhead position; a user eye position; and a user eye pair position.
 7. Thesystem of claim 1, wherein the user perspective further comprises anoblique viewpoint, wherein the render plane corresponds to the obliqueviewpoint, wherein the render plane has a first oblique render planeangle, and wherein an additional render plane corresponds to the obliqueviewpoint wherein the additional render plan has a second oblique renderplane angle.
 8. The system of claim 1, wherein the 3D scene renders instereo and the at least one display surface is a stereo 3D displaysurface.
 9. The system of claim 1, wherein the at least one processor isfurther configured to: present, via the at least one display, the 3Dscene according to one of the first field of view and first view volumeand the second field of view and second view volume.
 10. The system ofclaim 1, wherein the first virtual viewpoint is in one of open space andinner space.
 11. A non-transitory computer readable memory mediumstoring program instructions for presenting a view based on a virtualviewpoint in a three dimensional (3D) scene, wherein the programinstructions are executable by a processor to: determine a userviewpoint based on tracking of a user position, wherein the userposition comprises position and orientation in physical space; determinea user perspective relative to at least one display surface of at leastone display, wherein the user perspective comprises a mapping betweenangle and orientation of the at least one display surface and a renderplane to the user viewpoint; render and display the 3D scene within avirtual space based on a projecting in virtual space to the renderplane, wherein the render plane has a correlation to the position andorientation of the at least one display, and wherein the correlation isbased on the user perspective; determine first and second virtualviewpoints corresponding to first and second positions, angles andorientations in the virtual space, wherein the first and second virtualviewpoints are based on first and second positions, angles andorientations of at least a portion of a hand of the user in the physicalspace without use of hand-held tools; and establish first and secondfields of view and first and second view volumes of the 3D scene basedon the first and second virtual viewpoints.
 12. The non-transitorycomputer readable memory medium of claim 11, wherein the userperspective is based on a first user eypoint and the user position. 13.The non-transitory computer readable memory medium of claim 12, whereinto determine the user perspective, the program instructions are furtherexecutable to: determine a first user eyepoint based on user position;and correlate user position to a position of the at least one displaysurface, wherein the correlation is relative to an angle and orientationof the at least one display surface.
 14. The non-transitory computerreadable memory medium of claim 11, wherein the program instructions arefurther executable to: store the first and second fields of view and thefirst and second view volumes.
 15. The non-transitory computer readablememory medium of claim 11, wherein the first and second positions,angles, and orientations of the at least a portion of the hand of theuser are in open space.
 16. The non-transitory computer readable memorymedium of claim 11, wherein the user position comprises at least one of:a user head position; a user eye position; and a user eye pair position.17. The non-transitory computer readable memory medium of claim 11,wherein the user perspective further comprises an oblique viewpoint,wherein the render plane corresponds to the oblique viewpoint, whereinthe render plane has a first oblique render plane angle, and wherein anadditional render plane corresponds to the oblique viewpoint wherein theadditional render plan has a second oblique render plane angle.
 18. Thenon-transitory computer readable memory medium of claim 11, wherein the3D scene renders in stereo and the at least one display surface is astereo 3D display surface.
 19. The non-transitory computer readablememory medium of claim 11, wherein the program instructions are furtherexecutable to: present, via the at least one display, the 3D sceneaccording to one of the first field of view and first view volume andthe second field of view and second view volume.
 20. A method forpresenting a view based on a virtual viewpoint in a three dimensional(3D) scene, the method comprising: determining a user viewpoint based ontracking of a user position, wherein the user position comprisesposition and orientation in physical space; determining a userperspective relative to at least one display surface of at least onedisplay, wherein the user perspective comprises a mapping between angleand orientation of the at least one display surface and a render planeto the user viewpoint; rendering and display the 3D scene within avirtual space based on a projecting in virtual space to the renderplane, wherein the render plane has a correlation to the position andorientation of the at least one display, and wherein the correlation isbased on the user perspective; determining first and second virtualviewpoints corresponding to first and second positions, angles andorientations in the virtual space, wherein the first and second virtualviewpoints are based on first and second positions, angles andorientations of at least a portion of a hand of the user in the physicalspace without use of hand-held tools; and establishing first and secondfields of view and first and second view volumes of the 3D scene basedon the first and second virtual viewpoints.